This paper presents SimVTP: a Simple Video-Text Pretraining framework via masked autoencoders. We randomly mask out the spatial-temporal tubes of input video and the word tokens of input text and then feed them into a unified autencoder to reconstruct the missing pixels and words. Our SimVTP has several properties: 1) Thanks to the unified autoencoder, SimVTP reconstructs the masked signal of one modality with the help from another modality, which implicitly learns the cross-modal alignment between video tubes and text tokens. 2) SimVTP not only benefits from a high video masking ratio (e.g. 90%) due to the temporal redundancy of video, but also needs a high text masking ratio (e.g. 75%), which is much higher than BERT (e.g. 15%), to achieve optimal performance. This is because the aid of video modality makes text reconstruction less challenging, which thus needs a higher mask ratio to make the pretext harder for useful feature learning. 3) Equipping SimVTP with video-text contrastive learning (VTC) and video-text matching (VTM), which are two commonly used cross-modal training strategies, could further improve the transferable performance significantly. 4) SimVTP is dataefficent, e.g., pre-training only on 10% data of WebVid-2M, SimVTP achieves surprisingly good results (43.8 R@1) on MSRVTT, which is far above recent state-of-the-art methods pre-trained on both CC3M and WebVid-2M. We transfer our pre-trained model to various downstream tasks and achieve superior performance. The codes and models will be released at https://github.com/mayuelala/SimVTP.

Antimicrobial resistance is one of the biggest health problem, especially in the current period of COVID-19 pandemic. Due to the unique membrane-destruction bactericidal mechanism, antimicrobial peptide-mimetic copolymers are paid more attention and it is urgent to find more potential candidates with broad-spectrum antibacterial efficacy and low toxicity. Artificial intelligence has shown significant performance on small molecule or biotech drugs, however, the higher-dimension of polymer space and the limited experimental data restrict the application of existing methods on copolymer design. Herein, we develop a universal random copolymer inverse design system via multi-model copolymer representation learning, knowledge distillation and reinforcement learning. Our system realize a high-precision antimicrobial activity prediction with few-shot data by extracting various chemical information from multi-modal copolymer representations. By pre-training a scaffold-decorator generative model via knowledge distillation, copolymer space are greatly contracted to the near space of existing data for exploration. Thus, our reinforcement learning algorithm can be adaptive for customized generation on specific scaffolds and requirements on property or structures. We apply our system on collected antimicrobial peptide-mimetic copolymers data, and we discover candidate copolymers with desired properties.

Scene understanding is an essential and challenging task in computer vision. To provide the visually fundamental graphical structure of an image, the scene graph has received increased attention due to its powerful semantic representation. However, it is difficult to draw a proper scene graph for image retrieval, image generation, and multi-modal applications. The conventional scene graph annotation interface is not easy to use in image annotations, and the automatic scene graph generation approaches using deep neural networks are prone to generate redundant content while disregarding details. In this work, we propose SGDraw, a scene graph drawing interface using object-oriented scene graph representation to help users draw and edit scene graphs interactively. For the proposed object-oriented representation, we consider the objects, attributes, and relationships of objects as a structural unit. SGDraw provides a web-based scene graph annotation and generation tool for scene understanding applications. To verify the effectiveness of the proposed interface, we conducted a comparison study with the conventional tool and the user experience study. The results show that SGDraw can help generate scene graphs with richer details and describe the images more accurately than traditional bounding box annotations. We believe the proposed SGDraw can be useful in various vision tasks, such as image retrieval and generation.

Deep learning methods have contributed substantially to the rapid advancement of medical image segmentation, the quality of which relies on the suitable design of loss functions. Popular loss functions, including the cross-entropy and dice losses, often fall short of boundary detection, thereby limiting high-resolution downstream applications such as automated diagnoses and procedures. We developed a novel loss function that is tailored to reflect the boundary information to enhance the boundary detection. As the contrast between segmentation and background regions along the classification boundary naturally induces heterogeneity over the pixels, we propose the piece-wise two-sample t-test augmented (PTA) loss that is infused with the statistical test for such heterogeneity. We demonstrate the improved boundary detection power of the PTA loss compared to benchmark losses without a t-test component.

本文回顾了AIM 2022上压缩图像和视频超级分辨率的挑战。这项挑战包括两条曲目。轨道1的目标是压缩图像的超分辨率，轨迹〜2靶向压缩视频的超分辨率。在轨道1中，我们使用流行的数据集DIV2K作为培训，验证和测试集。在轨道2中，我们提出了LDV 3.0数据集，其中包含365个视频，包括LDV 2.0数据集（335个视频）和30个其他视频。在这一挑战中，有12支球队和2支球队分别提交了赛道1和赛道2的最终结果。所提出的方法和解决方案衡量了压缩图像和视频上超分辨率的最先进。提出的LDV 3.0数据集可在https://github.com/renyang-home/ldv_dataset上找到。此挑战的首页是在https://github.com/renyang-home/aim22_compresssr。

基于匹配的方法，尤其是基于时空记忆的方法，在半监督视频对象分割（VOS）中明显领先于其他解决方案。但是，不断增长和冗余的模板特征导致推断效率低下。为了减轻这一点，我们提出了一个新型的顺序加权期望最大化（SWEM）网络，以大大降低记忆特征的冗余。与以前仅检测帧之间特征冗余的方法不同，Swem通过利用顺序加权EM算法来合并框架内和框架间的相似特征。此外，框架特征的自适应权重具有代表硬样品的灵活性，从而改善了模板的歧视。此外，该提出的方法在内存中保留了固定数量的模板特征，从而确保了VOS系统的稳定推理复杂性。对常用的戴维斯和YouTube-VOS数据集进行了广泛的实验，验证了SWEM的高效率（36 fps）和高性能（84.3 \％$ \ Mathcal {J} \＆\ Mathcal {F} $代码可在以下网址获得：https：//github.com/lmm077/swem。

使用增强现实（AR）用于导航目的，这表明在手术手术过程中协助医生有益。这些应用通常需要知道外科手术工具和患者的姿势，以提供外科医生在任务执行过程中可以使用的视觉信息。现有的医学级跟踪系统使用放置在手术室内的红外摄像头（OR）来识别感兴趣的对象附加并计算其姿势的复古反射标记。一些市售的AR头式显示器（HMD）使用类似的摄像头进行自定位，手动跟踪和估算对象的深度。这项工作提出了一个使用AR HMD的内置摄像机来准确跟踪复古反射标记的框架，例如在手术过程中使用的标记，而无需集成任何其他组件。该框架还能够同时跟踪多个工具。我们的结果表明，横向翻译的准确度为0.09 +-0.06毫米，可以实现标记的跟踪和检测，纵向翻译的0.42 +-0.32 mm，绕垂直轴旋转的0.80 +-0.39 ver。此外，为了展示所提出的框架的相关性，我们在手术程序的背景下评估了系统的性能。该用例旨在在骨科过程中复制K-Wire插入的场景。为了进行评估，为两名外科医生和一名生物医学研究人员提供了视觉导航，每次都进行了21次注射。该用例的结果提供了与基于AR的导航程序报告的相当精度。

全球综合合作对于限制全球温度的升高至关重要，同时继续经济发展，例如减少严重的不平等或实现长期经济增长。与N战略代理进行缓解气候变化的长期合作提出了一个复杂的游戏理论问题。例如，代理商可以谈判并达成气候协议，但是没有中央权力可以执行遵守这些协议。因此，设计谈判和协议框架以促进合作，允许所有代理人达到其个人政策目标并激励长期遵守，这一点至关重要。这是一个跨学科的挑战，要求在机器学习，经济学，气候科学，法律，政策，道德和其他领域进行研究人员之间的合作。特别是，我们认为机器学习是解决该领域复杂性的关键工具。为了促进这项研究，在这里，我们介绍了一个多区域综合评估模型，模拟全球气候和经济，可用于设计和评估不同谈判和协议框架的战略成果。我们还描述了如何使用多代理增强学习来使用水稻N训练理性剂。该框架是全球气候合作的基础，这是一个工作组协作和气候谈判和协议设计的竞争。在这里，我们邀请科学界使用Rice-N，机器学习，经济直觉和其他领域知识来设计和评估其解决方案。更多信息可以在www.ai4climatecoop.org上找到。

视频文本预训练（VTP）旨在从大规模的网络视频中学习可转移的代表。迄今为止，几乎所有现有的VTP方法都仅限于基于检索的下游任务，例如视频检索，而它们在基于本地化的任务（例如时间基础）上的转移潜力不足。在本文中，我们实验分析并证明了当前VTP方法与本地化任务的不相容性，并提出了一种新颖的面向定位的视频文本预训练框架，称为LocvTP。具体而言，我们执行细粒对比度对准作为通过剪贴字对数发现方案对粗粒粒度的补充。为了进一步增强学习功能的时间推理能力，我们提出了一个上下文投影头和暂时意识的对比损失，以感知上下文关系。对六个数据集的四个下游任务进行的广泛实验表明，我们的LOCVTP在基于检索和基于本地化的任务上都达到了最先进的性能。此外，我们进行了全面的消融研究和彻底的分析，以探索最佳的模型设计和培训策略。

在对抗文献中，鲁棒性和准确性之间的权衡得到了广泛的研究。尽管仍然有争议，但普遍的观点是，从经验或理论上，这种权衡是固有的。因此，我们在对抗训练中挖掘了这种权衡的起源，发现它可能源于不当定义的可靠错误，该错误施加了局部不变性的诱导偏见 - 对平稳性的过度校正。鉴于此，我们主张采用局部模棱两可来描述健壮模型的理想行为，从而导致自洽的强大错误称为得分。根据定义，得分有助于稳健性与准确性之间的对帐，同时仍通过稳健优化处理最坏情况的不确定性。通过简单地将KL差异替换为距离指标的变体，得分可以有效地最小化。从经验上讲，我们的模型在AutoAttact下的强力板上实现了最高的性能。此外，得分提供了指导性见解，以解释在健壮模型上观察到的过度拟合现象和语义输入梯度。代码可在https://github.com/p2333/score上找到。